Digital watermarking of content is very well known. Herein, the term “content” or “item of content” refers to data comprising one or more of video data, image data, audio data, media or multimedia data, text data, or any other form of content data.
There are many methods for performing digital watermarking of content but, in general, they all involve adding a watermark to an item of content. This involves embedding, or adding, watermark symbols (or a watermark codeword or payload data) into the original item of content to form a watermarked item of content. The watermarked item of content can then be distributed to one or more users (or recipients or receivers). The method used for adding a watermark to an item of content depends on the intended purpose of the watermark. Some watermarking techniques are designed to be “robust”, in the sense that the embedded watermark can be successfully decoded even if the watermarked item of content has undergone subsequent processing (be that malicious or otherwise). Some watermarking techniques are designed to be “fragile”, in the sense that the embedded watermark cannot be successfully decoded if the watermarked item of content has undergone subsequent processing or modification. Some watermarking techniques are designed such that the difference between the original item of content and the watermarked item of content is substantially imperceptible to a human user (e.g. the original item of content and the watermarked item of content are visually and/or audibly indistinguishable to a human user). Other criteria for how a watermark is added to an item of content exist.
Fingerprint watermarking is increasingly being used to trace or identify the source of (or a party/device involved in) the unauthorized distribution or publication or release of content. For this type of watermarking process, the watermarked content provided to a receiver contains a watermark specific to that receiver (i.e. the watermark carries or encodes a payload or codeword specific to that receiver and identifying that receiver). Each of the receivers receives a copy of the original item of content with their respective watermark embedded therein. Then, if an unauthorized copy of the item of content is located, the watermark can be decoded from that item of content and the receiver that corresponds to the decoded watermark can be identified as the source of (or a party/device involved in providing/releasing) the unauthorized copy.
FIG. 1 of the accompanying drawings schematically illustrates a system 100 providing content delivery. The system 100 comprises a device 102 (a user device or a receiver), a content provider system 104 and a network 106. The content provider system 104 is arranged to provide content (or an item of content) to the device 102 via the network 106. The content could be delivered from the content provider system 104 to the device 102 by streaming the content, by downloading the whole or a part of the content, or by any other suitable mechanism.
The network 106 may be any kind of network suitable for transmitting or communicating data between the content provider system 104 and the device 102 and, in particular, for communicating content from the content provider system 104 to the device 102. For example, the network 106 could comprise one or more of a local area network, a wide area network, a metropolitan area network, the internet, a wireless communications network, a cable network, a digital broadcast network, a satellite communication network, a telephone network, etc. The content provider system 104 may then communicate with the device 102 over the network 106 via any suitable communication mechanism/protocol in order to communicate data (such as content) between the content provider system 104 and the device 102.
The content provider system 104 may be any system that is suitable for communicating data with the device 102 via the network 106. The content provider system 104 comprises one or more processors 110, a memory 114, a network interface 112 and a data store 108. The network interface 112 is arranged to interface with the network 106 to enable the content provider system 104 to communicate with the network 106 (so that the content provider system 104 can then communicate with the device 102 via the network 106). The data store 108 may be a database or other storage mechanism/structure, and may store one or more items of content (or, as shall be described shortly, one or more sections of one or more items of content) that may be requested by the device 102 and that the content provider system 104 may then communicate to the device 102 via the network 106. The processor(s) 110 control the operation of the content provider system 104 and execute logic to carry out functionality of the content provider system 104, and may, as part of this processing, make use of the memory 114 to store and/or process data as is well-known in the art.
As examples, the content provider system 104 could be (part of) a headend system of a digital broadcast system (in which case the network 106 could comprise a terrestrial broadcast network or a satellite broadcast network) or the content provider system 104 could be (part of) a headend system of a cable network system (in which case the network 106 could comprise a cable network). The content provider system 104 could comprise one or more servers for transmitting, or providing access to, data over the internet (in which case the network 106 may comprise the internet). However, it will be appreciated that the content provider system 104 may take other forms instead and that the architecture illustrated in FIG. 1 is purely exemplary—in particular, the content provider system 104 may have one or more other components (not shown); alternatively, one or more of the components of the content provider system 104 shown in FIG. 1 may form part of a separate system accessible by the content provider system 104 or may act as separate resources for use by the content provider system 104.
The device 102 may be any system or apparatus that is suitable for communicating data with the content provider system 104 over the network 106. The device 102 comprises a network interface 120, one or more processors 116 and a memory 118. The network interface 120 is arranged to interface with the network 106 to enable the device 102 to communicate with the network 106 (so that the device 102 can then communicate with the content provider system 104 via the network 106). The processor(s) 116 control the operation of the device 102 and execute logic to carry out functionality of the device 102 and may, as part of this processing, make use of the memory 118 to store and/or process data as is well-known in the art.
The device 102 is arranged to execute a content player 122 (or a media rendering device/application/module). The content player 122 may process and output/render content that the device 102 has received from the content provider system 104 via the network 106—this output may be via one or more output devices not shown in FIG. 1 (such as speakers, displays/screens, etc.) of, or coupled to, the device 102.
The content player 122 may be implemented as hardware and/or as software executing on the processor(s) 116. As an example, the content player 122 may be a software media player (i.e. an audio and/or video player) such as QuickTime, Windows Media Player, RealPlayer, etc.—this media player may be executed by the processor(s) 116, either as a standalone application or as a component embedded within another application (such as when a browser displays a webpage that includes a media player). The content player 122 may make use of dedicated content rendering hardware (such as decoding/decompression hardware).
Additionally, if the system 100 is to provide conditional access (CA) and/or digital rights management (DRM) functionality, the device 102 may be arranged to make use of one or more hardware devices (such as smartcards) and/or security software modules to carry out the CA and/or DRM functionality, as is well-known in the art.
As examples, the device 102 may comprise one or more of a set-top-box, a personal computer, a mobile telephone, a games console, a television, a radio, etc., but it will be appreciated that the device 102 may take other forms instead and that the architecture illustrated in FIG. 1 is purely exemplary—in particular, the device 102 may have one or more other components (not shown); alternatively, one or more of the components of the device 102 shown in FIG. 1 may form part of a separate system accessible by the device 102 or may act as separate resources for use by the device 102.
Although a single content provider system 104, a single network 106 and a single device 102 are illustrated in FIG. 1, it will be appreciated that the system 100 could comprise multiple content provider systems 104, multiple networks 106 and multiple devices 102, and that FIG. 1 has been simplified for ease of illustration. In particular, each content provider system 104 may communicate with multiple devices 102 over one or more networks 106, and each device 102 may communicate with multiple content provider systems 104 over one or more networks 106.
As is known in the art, the network 106 may comprise one or more caches 124, or one or more other network nodes, which may each store some or all of the same data as stored in the data store 108, or additional data not stored in the data store 108. When the device 102 submits a request for content to the content provider system 104 via the network 106, one or more of the caches 124 may be storing the requested data (in addition to, or in place of, the content provider system 104 storing that content)—when this happens, it may be more efficient (from the perspective of the content provider system 104), quicker and more bandwidth efficient for the requested content (or part of the requested content) to be provided to the device 102 from one or more of these caches 124 instead of from the content provider system 104 itself, in which case the network 106 may be arranged to provide the requested content (or that part of the requested content) to the device 102 from these caches 124 instead of from the content provider system 104 itself. Thus, the network 106, with its caches 124, along with the content provider system 104, may be viewed as forming a content distribution system 126. The device 102 may therefore interact with the content distribution system 126 to request and access content, and the content distribution system 126 may respond to the request by providing, or communicating, the requested content to the device 102. If caches 124 are not used, then the content distribution system 126 may be viewed simply as the content provider system 104 together with the network 106, or even just as the content provider system 104. The rest of this description shall refer to content (or sections of content) being stored at, and provided by, the content provider system 104, but it will be appreciated that content (or sections of content) may be stored at, and provided by, one or more caches 124 instead of, or in addition to, the content provider system 104, so that, in general, the content (or sections of content) may be stored at, and provided by, the content distribution system 126.
European patent application EP2429189 (the entire disclosure of which is incorporated herein by reference) discloses how to use a system such as the system 100 of FIG. 1 to carry out adaptive content delivery with watermarking (or fingerprinting). FIGS. 2 and 3 of the accompanying drawings depict how the system 100 of FIG. 1 may be arranged to carry out adaptive content delivery with watermarking (or fingerprinting) based on the methods and systems set out in EP2429189.
FIG. 2 schematically illustrates how an item of content 200 may be viewed as, or divided/separated into, a number of sections 202 (or content chunks). In FIG. 2, there are m sections 202 labelled S1, S2, S3, S4, . . . , Sm. The sections 202 are usually non-overlapping and, when concatenated together in order (i.e. in their time-based ordering), form the item of content 200. Each section 202 may contain, or represent, a few seconds of content or a number of frames/fields of audio/video (or some other time-based quantity of content). The sections 202 may be of the same length, or may be of different lengths.
For one or more of the sections 202, there may be multiple versions (or variants) 204 of that section 202. A version 204 of a section 202 may be generated by encoding a copy of that section 202 (e.g. data compression, formatting, etc.) to achieve a corresponding quality level or a corresponding bandwidth requirement—the lower the quality level, the less bandwidth is required to communicate the section version 204 across the network 106 to the device 102. These different quality levels (or bandwidth characteristics) can be achieved by, for example, changing the resolution of video/image data (higher qualities may use higher resolutions), changing quantisation or sampling levels (higher qualities corresponding to less quantisation), etc., as is well-known in the art. In FIG. 2, for section S1, there are n versions 204 of that section S1 labelled S1Q1, S1Q2, . . . ,S1Qn, (where n≥1). Herein, a label “SiQj” indicates the i-th section Si encoded to a quality level A version 204 of a section 202 may be generated by watermarking a copy of that section 202 to embed a corresponding message/payload/symbol therein. In FIG. 2, for section S2, there are t (where t≥2) watermarked versions 204 of that section S2 labelled S2M1, S2M2, . . . , S2Mt, that all have their own respective (relatively different) watermarks embedded therein, so that they are all distinct versions of the section S2. Herein, a label “SiMk” indicates the k-th watermarked version of the i-th section Si. The encoding and watermarking may be combined, so that a version 204 of a section 202 may be generated by watermarking a copy of that section 202 that has been encoded in a manner to achieve a corresponding quality level or a corresponding bandwidth requirement, in order to embed a corresponding message/payload/symbol therein. In FIG. 2, for section S4, there are n quality levels for that section S4, and for each one of those n quality levels, there are t watermarked versions 204—these are labelled S4Q1M1, . . . , S4QnMt (where n≥1 and t≥2). The watermarks used for any given quality level are different from each other. Herein, a label “SiQjMk” indicates the k-th watermarked version of the i-th section Si encoded at quality level Qj. It will be appreciated that for different quality levels Qj1 and Qj2, the number of different watermarked versions of a section 202 encoded at those quality levels may be different from each other, and the watermarks embedded for one quality level may differ from the watermarks embedded for another quality level. Different sections 202 may have different numbers of versions 204 generated using (a) different numbers of quality levels (and the actual quality levels used may vary from section 202 to section 202) and/or (b) different numbers of watermarks (and the actual watermarks embedded may vary from section 202 to section 202).
Some sections 202 may only have a single version 204 (which could be the same as, or different from, that original section 202).
Thus, for each of the plurality of sections 202 of the item of content 200, the content provider system 104 is arranged to provide one or more versions 204 of that section. To achieve the fingerprint watermarking, there is at least one section 202 for which the content provider system 104 is arranged to provide a plurality of differently watermarked versions of that section 202 (be they versions SiMk not encoded to different quality levels or versions SiQjMk that are encoded to different quality levels).
The content provider system 104 may store the section versions 204 in the data store 108 (possibly along with the original content sections 202). As mentioned above, some or all of the section versions 204 may be stored in one or more of the caches 124.
If limited bandwidth is available for communicating content to the device 102, this may be detected by the content player 122 and the content player 122 can then choose to request a section version 204 with a lower bandwidth characteristic (i.e. a lower quality level). If/when more bandwidth becomes available for communicating content to the device 102, this may be detected by the content player 122 and the content player 122 can choose to request a section version 204 with a higher bandwidth characteristic (i.e. a higher quality level). These requests can be satisfied by the content provider system 104 providing the device 102 with the requested section version 204 encoded at a quality level appropriate to the requested bandwidth characteristic. Similarly, if limited bandwidth is available for communicating content to the device 102, this may be detected by the content provider system 104 and the content provider system 104 can then choose to provide to the device 102 a section version 204 with a lower bandwidth characteristic (i.e. a lower quality level). If/when more bandwidth becomes available for communicating content to the device 102, this may be detected by the content provider system 104 and the content provider system 104 can choose to provide to the device 102 a section version 204 with a higher bandwidth characteristic (i.e. a higher quality level). For example, the content player 122 may need to output/render content sections 202 to a user at a particular frequency. If the content player 122 detects that it is not receiving sections versions 204 from the content provider system 104 in time to be able to output/render the section versions 204 at that frequency, then the content player 122 may issue subsequent requests that are for section versions 204 at a lower quality than the content player 122 had previously been requesting. Similarly, if the content player 122 detects that it is has been receiving sections versions 204 from the content provider system 104 well in advance of the time at which those section versions 204 needed to be output/rendered, then the content player 122 may issue subsequent requests that are for section versions 204 at a higher quality than the content player 122 had previously been requesting.
To achieve fingerprint watermarking, a receiver (or a user or a device 102, or a group of receivers, users or devices 102) is provided access to a corresponding specific sequence (or set or group) of watermarked section versions 204 (be they versions SiMk or SiQjMk). This specific sequence is different from sequences of watermarked section versions 204 used by other receivers. In particular, if there are W receivers in a population of receivers, then the w-th receiver (1≤w≤W) may be associated with a sequence of watermarked section versions Si(w,1)Mk(w,1), Si(w,2)Mk(w,2), . . . , Si(w,P)Mk(w,P), where there are P watermarked section versions 204 in the specific sequence associated with the w-th receiver, and for 1≤p≤P, the p-th section version 204 in the specific sequence is Si(w,p)Mk(w,p) for some indices i(w,p) and k(w,p). This sequence of watermarked section versions Si(w,1)Mk(w,i), Si(w,2)Mk(w,2), . . . , Si(w,P)Mk(w,P), (or the sequence of index pairs (i(w,1),k(w,1)), (i(w,2),k(w,2)), . . . , (i(w,P),k(w,P))) is unique to the w-th receiver. It will be appreciated that section versions using different quality levels could be involved in the above sequence too. The device 102, upon receiving the section versions 204 from the content provider system 104, may combine (e.g. concatenate) the received section versions 204 to form a version of the original item of content 200. As this version of the original item of content 200 will use the specific sequence of watermarked section versions 204, it will be specific to that receiver associated with that device 104. Therefore, if that version of the original item of content 200 is distributed or released in an unauthorised manner, a watermark decoder can be used to identify that that specific sequence of watermarked section versions 204 was used to form that version of the original item of content 200, and hence identify that particular receiver.
As an example, the content provider system 104 may associate each receiver with a corresponding P-bit identification number—let the i-th bit of the identification number be b(i) (1≤i≤P). If P sections S1, . . . ,SP are available and each of these sections 202 has corresponding differently watermarked sections versions SiM0 and SiM1 (1≤i≤P) (i.e. a version 204 watermarked with a symbol representing a “0” and a version 204 watermarked with a symbol representing a “1”), then a receiver's P-bit identification number may be associated with a sequence of watermarked sections corresponding to that P-bit identification number—in particular, the i-th section version 204 in that sequence will be SiMb(i). For example, if P=6 and the identification number associated with a particular receiver is 110101, then that receiver may be provided access to watermarked section versions S1M1, S2M1, S3M0, S4M1, S5M0 and S6M1.
The use of an identification number in this way is not essential. The sequence of watermarked section versions 204 associated with a particular receiver could be randomly determined/generated. The sequence of watermarked section versions 204 need not be of fixed length (length P above) but could, for example, keep increasing in size as more and more section versions are provided to a receiver. The sequence of watermarked section versions 204 may be predetermined for a receiver or may be dynamically generated as and when the section versions 204 are being provided to the receiver. It will be appreciated that other mechanisms for determining a specific sequence of watermarked section versions 204 associated with a particular receiver could be used. However, in general, the content provider system 104 determines (or generates or calculates) the specific sequence of watermarked section versions 204 to associate with a particular receiver and may store data identifying (or from which can be calculated) that specific sequence of watermarked section versions 204 (e.g. the sequence of index pairs mention above, or a seed value used for randomly choosing a watermarked section version 204).
FIG. 3 schematically illustrates a process and architecture for the content provider system 106 to prepare the item of content 200 for adaptive content delivery with watermarking (or fingerprinting). This may be carried out by the processor(s) 110 executing one or more software processes; additionally or alternatively, this may be carried out by one or more hardware components of the content provider system 106 (not shown in FIG. 1).
The original item of content 200 is either received by an input 300 or is generated by a content generator 302. This original item of content 200 may contain a watermark (e.g. a watermark identifying information such as an owner of copyright in the content 200 or other metadata concerning the content 200)—such a watermark may have been embedded into the content 200 by another system (not shown in the figures).
The item of content 200 is provided to content encoders 304. In FIG. 3, two content encoders 304(1) and 304(2) are illustrated, and each content encoder 304 is arranged to encode the item of content 200 at a respective quality encoding level different from the quality encoding level used by the other content encoders 304. It will, of course, be appreciated that more than two content encoders 304 could be used, each arranged to encode the item of content 200 at its own respective quality encoding level different from the quality encoding levels used by the other content encoders 304. It will, additionally, be appreciated that a single content encoder 304 could be used so that, instead of encoding the item of content 200 at different quality levels in parallel (as is shown in FIG. 2), the item of content 200 may be encoded by the single content encoder 304 at a first quality level, then the item of content 200 may be encoded by the same content encoder 304 at a second quality level, and so on. A mixture of such encoder configurations is possible.
Each of these encoded items of content is split (or divided or separated) into a number of section versions 204 by respective section generators 306. As for the content encoders 304, in FIG. 3 two section generators 306(1) and 306(2) are illustrated, and each section generator 306 is arranged to generate section versions 204 from the encoded item of content 200 provided by a corresponding content encoder 304. It will, of course, be appreciated that more than two section generators 306 could be used, for example if more than two content encoders 304 are used. It will, additionally, be appreciated that a section generator 306 may generate section versions 204 out of encoded items of content 200 provided by multiple content encoders 304—for example, a single section generator 306 could be used in FIG. 3, where this single section generator 306 is arranged to generate a first set of section versions 204 from the encoded item of content 200 output by the first content encoder 304(1) and to then generate a second set of section versions 204 from the encoded item of content 200 output by the second content encoder 304(2), and so on. A mixture of such section generator configurations is possible.
A time alignment module 308 coordinates the generation of the section versions 204 by the section generators 306 so that the section versions 204 generated from respective encoded items of content 200 (output by the content encoders 304) are time-aligned, i.e. so that two section versions SiQq1 and SiQq2 for the i-th section 202 of the original item of content 200 encoded at different quality levels q1 and q2 will start at the same point within the original item of content 200 and will end at the same point within the original item of content 200. In this way, a version of the item of content 200 can be formed by selecting, for each section 202 of the item of content 200, a respective section version 204 and then concatenating those section versions 204 in order—different section versions 204 corresponding to the same section 202 of the item of content 200 can be interchanged as they are time-aligned. In other words, seamless switching between section versions 204 with different encoding quality can be carried out.
Each of the section versions 204 output by the section generators 306 is then supplied to a modifier module 307. The modifier module 307 is arranged to modify or process a section version 204 that it has received in order to create a modified section version 204. This modification may be, for example, by watermarking the received section version 204 so as to embed a corresponding symbol or payload into the received section version 204—this may be achieved by know watermarking techniques. In FIG. 3, four modifier modules 307(1)-307(4) are illustrated: a first modifier module 307(1) receives the section versions 204 output by the first section generator 306(1) and embeds a first symbol into those section versions 204 to output a first set of final section versions 204; a second modifier module 307(2) receives the section versions 204 output by the first section generator 306(1) and embeds a second symbol into those section versions 204 to output a second set of final section versions 204; a third modifier module 307(3) receives the section versions 204 output by the second section generator 306(2) and embeds the first symbol into those section versions 204 to output a third set of final section versions 204; and a fourth modifier module 307(4) receives the section versions 204 output by the second section generator 306(2) and embeds the second symbol into those section versions 204 to output a fourth set of final section versions 204. In this particular configuration, for an initial section Si of the original item of content 200, there will be generated four corresponding sections versions SiQ1M1, SiQ1M2, SiQ2M1, SiQ2M2. It will, of course, be appreciated that a different number of modifier modules 307 could be used. For example, if t watermarked versions of a section output by a section generator 306 are to be generated, then t modifier modules 307 could be implemented and coupled to that section generator 306. It will, additionally, be appreciated that a modified module 307 may be arranged to modify a section in a first manner to output a first section version 204 (e.g. by embedding a first watermark symbol) and may then modify that initial received section in a second manner to output a second section version 204 (e.g. by embedding a second different watermark symbol), and so on. A modifier module 307 may be arranged to receive sections generated by multiple section generators 306. A mixture of such modified module configurations is possible.
It will be appreciated that the functionality provided by the content encoders 304 and/or the section generators 306 and/or the modifier modules 307 may be implemented in whole or in part in together by a single module, rather than by separate modules as shown in FIG. 3.
The ordering in which the processing is illustrated in FIG. 3 may be changed. For example, the content sections 202 may be generated initially by one or more section generators 306 from the original item of content 200, and these sections 202 may then be passed to content encoders 304 which then encode the sections 202 to generate the section versions 204 and these may then be modified by modifier modules 307. Similarly, different watermarked versions of the original item of content 200 may be generated by one or more modifier modules 307, these modified items of content may then be encoded to different quality levels by respective content encoders 306, following which the sections versions 204 may be generated by one or more section generators 306 from the encoded modified items of content. Other arrangements are, of course, possible to result in the generation of the section versions 204.
The generated section versions 204 are stored in the data store 108.
Whilst FIG. 3 illustrates the generation of content versions 204 that are encoded at different quality levels and that have respective watermarks embedded therein (i.e. section versions SiQjMk), it will be appreciated that not all section versions 204 need to be watermarked, e.g. some of the section versions 204 output by the section generators 204 may be stored directly into the data store 108 in addition to, or possibly without being, passed to a modified module 307—in this way, section versions SiQj may be generated and stored in the data store 108. Similarly, it will be appreciated that not all section versions 204 need to be encoded to a particular quality level, e.g. a section generator 306 may receive the original item of content 200 instead of receiving an encoded item of content—in this way, section versions SiMk, or even just section versions Si, may be generated and stored in the data store 108.
It will be appreciated, therefore, that the arrangement shown in FIG. 3 is purely exemplary and that alternatives are possible to achieve the types of section version generation discussed above with reference to FIG. 2.
A content format module 310 may determine, for each of the section versions 204, a corresponding reference (or identifier) with which that section version 204 can be identified, accessed and retrieved from the data store. The content format module 310 may also generate additional/other metadata 312 that can ultimately be used by the content player 122 to generate the references that the content player 122 will need to request and retrieve the appropriate/desired section versions 204. For example, this metadata 310 may take the form of a playlist (i.e. a file separate from the content and containing links or references to locations, such as URLs, for the content versions 204)—this playlist may then be provided to the content player 122 so that the content player 122 can select one or more section versions 204 identified in the playlist and then request those selected section versions 204 from the content provider system 104. Alternatively, the content provider system 104 may provide a content file to the device 102 containing one or more section versions 204, and the metadata 310 may take the form of a manifest (i.e. a portion of the content file provided to the device 102) that identifies where, within the content file, the section versions 204 are located. For performing fingerprint watermarking, the section versions 204 identified in the playlist or manifest provided to a receiver will include the specific sequence of watermarked section versions 204 associated with that receiver.
The structure of the playlist or manifest may sometimes have to depend on the content format. A new content format may, therefore, require updating of existing playlist generators (within the content format module 310) in order to be able to support that new content format. Moreover, some content formats do not use or support the concept of a playlist (e.g. MicroSoft Smooth Streaming)—supporting such content formats within the framework of the above adaptive content delivery with watermarking (or fingerprinting) may then require a special client side agent, running on the device 102, to support such content formats. This involves additional effort by users running the device 102, and is more difficult for providers of content (e.g. those operating the content provider system 106) to develop and maintain, especially as new formats are developed or existing formats are changed. These problems are exacerbated when the network caches 124 and other network nodes are involved in delivering content to the device 102.